Building a Fingerprint Based Deduplication Detection and Elimination Scheme

As digital data is growing uncontrollably, the need for data reduction has become an important task in storage systems. For large scale data reduction, it is important to maximally detect and eliminate redundancy at low overheads. Data deduplication is a data reduction technique that reduces storage space by eliminating redundant data and only one instance of the data is retained on storage media. Delta compression is an efficient method for removing redundancy among non-duplicates but very similar data files and chunks. DARE is a deduplication aware, low overhead resemblance detection and elimination scheme for data reduction that uses duplicate-adjacency information for resemblance detection in a deduplication system. It also uses an improved super feature approach for further resemblance detection when DupAdj information is lacking or limited. DARE attained a superior performance of both throughput and data reduction efficiency among all resemblance detection approaches.